You could potentially use an authSwitchRequest (from deleted code above) if the plugin is unknown. The downside is that this is hard to test.

If the client presents an authentication plugin that doesn't match the plugin used by the server checkAuthPlugin will do the authSwitchRequest so I don't think this code will be reached during normal operation. It may be reached if someone bypasses the GRANT/REVOKE etc and changes mysql.user directly.

If open session success, but do auth fails, will there be a potential resource leak?

So, there are:

1. the default auth plugin (we set in the initial handshake packet)
2. the user's auth plugin, which stores in mysql.user authentication related column
3. the auth plugin from the client response packet

Which one is which? I'm a bit confused.

Pass *string as both input and output of a function is an idiom in C.
Go support multiple return, you don't need to do that.

Just a hint: open a new session may need to load the global variables (maybe not true when they're cached), and do it once again could slow down the connection speed.

I agree with your comment.

The cache is improved with sysvar cache, but it is still expensive. The current code will re-run validation on all of the sysvars. This can be improved though: in future we can cache the validated values in the sysvar cache, and when a session is initialized just copy the sysvar cache map to it.

The complicated case will be eliminating the need to call the 'init functions' (i.e. calling sv.SetSession() so that s.VarXXX is set.) I think we can remove a lot of SetSession() funcs by adding something like sv.ToBool(), sv.ToUint64().

Indeed, creating a new session for handshake is relatively heavy, indirectly leading to the occurrence of #44502.